There is a pervasive belief in the MIS research community that PLS has advantages over other techniques when analyzing small sample sizes or data with non-normal distributions. Based on these beliefs, major MIS journals have published studies using PLS with sample sizes that would be deemed unacceptably small if used with other statistical techniques. We used Monte Carlo simulation more extensively than previous research to evaluate PLS, multiple regression, and LISREL in terms of accuracy and statistical power under varying conditions of sample size, normality of the data, number of indicators per construct, reliability of the indicators, and complexity of the research model. We found that PLS performed as effectively as the other techniques in detecting actual paths, and not falsely detecting non-existent paths. However, because PLS (like regression) apparently does not compensate for measurement error, PLS and regression were consistently less accurate than LISREL. When used with small sample sizes, PLS, like the other techniques, suffers from increased standard deviations, decreased statistical power,and reduced accuracy. All three techniques were remarkably robust against moderate departures from normality, and equally so. In total, we found that the similarities in results across the three techniques were much stronger than the differences.
In the Foreword to an MIS Quarterly Special Issue on PLS, the senior editors for the special issue noted that they rejected a number of papers because the authors attempted comparisons between results from PLS, multiple regression, and structural equation modeling (Marcoulides et al. 2009). They raised several issues they argued had to be taken into account to have legitimate comparison studies, supporting their position primarily by citing three authors: Dijkstra (1983), McDonald(1996), and Schneeweiss (1993). As researchers interested in conducting comparison studies, we read the Foreword carefully, but found it did not provide clear guidance on how to conduct "legitimate" comparisons. Nor did our reading of Dijksta, McDonald, and Schneeweiss raise any red flags about dangers in this kind of comparison research. We were concerned that instead of helping researchers to successfully engage in comparison research, the Foreword might end up discouraging that type of work, and might even be used incorrectly to reject legitimate comparison studies. This Issues and Opinions piece addresses the question of why one might conduct comparison studies, and gives an overview of the process of comparison research with a focus on what is required to make those comparisons legitimate. In addition, we explicitly address the issues raised by Marcoulides et al., to explore where they might (or might not) come into play when conducting or evaluating this type of study.